Moving state estimating device

ABSTRACT

A moving state estimating device ( 100 ) includes a feature point candidate extracting part ( 10 ) configured to extract feature point candidates in an image taken at a first time point by an imaging sensor ( 2 ) mounted on a vehicle, a positional relation obtaining part ( 11 ) configured to obtain a positional relation of an actual position of each feature point candidate relative to the vehicle, a feature point selecting part ( 12 ) configured to select a feature point candidate as a feature point based on the positional relation of the feature point candidate, a corresponding point extracting part ( 13 ) configured to extract a corresponding point in an image taken by the imaging sensor ( 2 ) at a second time point corresponding to the selected feature point, and a moving state estimating part configured to estimate a moving state of the vehicle from the first time point to the second time point based on a coordinate of the selected feature point and a coordinate of the corresponding point.

TECHNICAL FIELD

The present invention relates to a moving state estimating device whichestimates a moving state of a moving body equipped with an image sensorbased on images taken by the imaging sensor.

BACKGROUND ART

Conventionally, a perimeter monitoring device for a moving body whichestimates a position and an orientation of a target vehicle by employinga monocular camera is known (see Patent Document 1).

This perimeter monitoring device for a moving body extracts four or morefeature points on a road surface from an image taken by the monocularcamera mounted on the target vehicle at a first time point while thetarget vehicle is moving. Then, while tracking the feature points, thedevice extracts a point which corresponds to each of these featurepoints on a road surface from an image taken by the monocular camera ata second time point. Then, the device derives homography based on thetwo-dimensional coordinates of the feature points and thetwo-dimensional coordinates of the corresponding points so that thedevice derives the relative variation between the position and theorientation of the monocular camera at the first time point and theposition and the orientation of the monocular camera at the second timepoint.

Consequently, this perimeter monitoring device for a moving body canestimate the position and the orientation of the target vehicle equippedwith the monocular camera.

[Patent Document 1] Japanese Patent Publication No. 2004-198211DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention

However, although Patent Document 1 describes that the perimetermonitoring device for a moving body extracts four or more feature pointsfrom a white line, a bump, a difference in level, or a pattern on theroad surface imaged by the monocular camera, it does not describe how toselect the feature points. Accordingly, the perimeter monitoring devicefor a moving body may extract biased feature points in terms of distanceor direction, and thus reduce the accuracy of estimation of the positionand the orientation of the target vehicle.

The perimeter monitoring device for a moving body may also extract apoint on a moving body such as other vehicles imaged by the monocularcamera as a feature point, and thus reduce the accuracy of estimation ofthe position and the orientation of the target vehicle.

In view of the above-mentioned problems, it is an object of the presentinvention to provide a moving state estimating device which achievesmore accurate estimation of a moving state of a moving body based onimages taken by an imaging sensor mounted on the moving body.

Means for Solving Problems

To achieve the object above, a moving state estimating device accordingto a first embodiment of the invention includes a feature pointcandidate extracting part configured to extract feature point candidatesin an image taken at a first time point by an imaging sensor mounted ona moving body, a positional relation obtaining part configured to obtaina positional relation of an actual position of each feature pointcandidate relative to the moving body, a feature point selecting partconfigured to select a feature point candidate as a feature point basedon the positional relation of the feature point candidate, acorresponding point extracting part configured to extract acorresponding point in an image taken by the imaging sensor at a secondtime point corresponding to the selected feature point, and a movingstate estimating part configured to estimate a moving state of themoving body from the first time point to the second time point based ona coordinate of the selected feature point and a coordinate of thecorresponding point.

According to a second embodiment of the invention, there is provided amoving state estimating device according to the first embodiment of theinvention, wherein the positional relation includes a distance betweenthe actual position of the feature point candidate and the moving body,a direction of the actual position of the feature point candidate asseen from the moving body, or a relative velocity of the actual positionof the feature point candidate with respect to the moving body.

According to a third embodiment of the invention, there is provided amoving state estimating device according to the first or the secondembodiment of the invention, wherein the feature point selecting part isconfigured to determine whether the feature point candidate isstationary or not, and to select the feature point from stationaryfeature point candidates.

According to a fourth embodiment of the invention, there is provided amoving state estimating device according to one of the first, thesecond, and the third embodiments of the invention, wherein the featurepoint selecting part is configured to select feature points so that adistance difference between an actual position of each feature point andthe moving body is greater than or equal to a predetermined value.

According to a fifth embodiment of the invention, there is provided amoving state estimating device according to one of the first, thesecond, the third, and the fourth embodiments of the invention, whereinthe feature point selecting part is configured to select feature pointsso that angles between directions of actual positions of each featurepoint as seen from the moving body are greater than or equal to apredetermined value.

Herein, “a feature point” stands for a coordinate point (a pixel) in animage for use in generation of homography, that is, a coordinate point(a pixel) selected from a feature point candidate under certainconditions. For example, the coordinate system defines a pixel at alower left corner of an image taken by an imaging sensor as its origin(0,0), while it defines a pixel which is immediately right of the originas the coordinate (1,0) and a pixel which is immediately above theorigin as the coordinate (0,1).

“A feature point candidate” stands for a coordinate point (a pixel)which can be a feature point, that is, an extractable pixel from animage under certain conditions, for example, a pixel whose luminancedifference with respect to the adjacent pixel is greater than or equalto a predetermined value or a center pixel of a pixel group whoseluminance pattern matches a predetermined luminance pattern, and so on.

“A corresponding point” stands for a coordinate point (a pixel) in animage other than the image in which the selected feature point exists,which coordinate point is included in a photographic subject designatedby a coordinate point (a pixel) selected as a feature point in the imagein which the selected feature point exists. For example, a correspondingpoint is extracted from the other image by matching a luminance patternof a pixel group in the other image in which the corresponding pointexists, with a luminance pattern of a pixel group in the image in whichthe selected feature point exists, for example at the center thereof.The luminance pattern may be a pattern based on other properties such asa color pattern.

“A positional relation” stands for information concerning a relativeposition between an actual position of a feature point candidate whichcan be a feature point and a moving body equipped with an imaging sensorwhich have taken an image. For example, the information may include adistance between an actual position of a feature point candidate whichcan be a feature point and a target vehicle, a direction of an actualposition of a feature point candidate as seen from a target vehicle, arelative velocity of an actual position of a feature point candidatewith respect to a target vehicle, and so on.

EFFECT OF THE INVENTION

According to the means above, it is possible for the present inventionto provide a moving state estimating device which achieves more accurateestimation of a moving state of a moving body based on images taken byan imaging sensor mounted on the moving body.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an embodiment of the moving stateestimating device according to the present invention.

FIG. 2A is a first illustration of a method for selecting a featurepoint from a plurality of feature point candidates.

FIG. 2B is a second illustration of the method for selecting a featurepoint from a plurality of feature point candidates.

FIG. 2C is a third illustration of the method for selecting a featurepoint from a plurality of feature point candidates.

FIG. 3 is a flowchart illustrating a flow of a moving state estimatingprocess.

EXPLANATION OF REFERENCE SIGNS

-   1 control device-   2 imaging sensor-   3 distance measuring sensor-   4 vehicle speed sensor-   5 steering angle sensor-   6 display device-   7 audio output device-   10 feature point candidate extracting part-   11 positional relation obtaining part-   12 feature point selecting part-   13 corresponding point extracting part-   14 moving state estimating part-   100 moving state estimating device-   D1-D3 space distance-   α, β angle between actual directions-   P1 first feature point-   P2 second feature point-   P3 third feature point-   V1 target vehicle-   V2 parked vehicle-   V3 moving vehicle

BEST MODE FOR CARRYING OUT THE INVENTION

With reference to the figures, the description of the best mode forcarrying out the present invention is given below.

Embodiments

FIG. 1 is a diagram illustrating an embodiment of the moving stateestimating device according to the present invention. The moving stateestimating device 100 is an in-car device which estimates a moving statesuch as translation or rotation of a vehicle. The moving stateestimating device 100 includes a control device 1 which is connected toan imaging sensor 2, a distance measuring sensor 3, a vehicle speedsensor 4, a steering angle sensor 5, a display device 6, and an audiooutput device 7 via an in-car LAN such as CAN (Controller Area Network)or others.

The control device 1 is a computer which includes a CPU (CentralProcessing Unit), a RAM (Random Access Memory), a ROM (Read OnlyMemory), and so on. For example, programs corresponding to a featurepoint candidate extracting part 10, a positional relation obtaining part11, a feature point selecting part 12, a corresponding point extractingpart 13, and a moving state estimating part 14 may respectively bestored in the ROM and the CPU executes a process corresponding to eachpart.

The imaging sensor 2 is a sensor for taking an image around a vehicle.For example, the imaging sensor 2 may be a camera with an image pickupdevice such as a CCD (Charge Coupled Device), a CMOS (ComplementaryMetal Oxide Semiconductor), or others. The imaging sensor 2 may also bean infrared camera capable of night photographing or a stereo cameracapable of measuring a distance to a photographic subject.

In an example, the imaging sensor 2 may be placed near a rearview mirrorin a vehicle interior with imaging direction fixed, and outputs to thecontrol device 1 images taken at a specified time interval.

The distance measuring sensor 3 is a sensor for obtaining informationconcerning a relative position between an object around a target vehicleand the target vehicle. For example, the distance measuring sensor 3 maybe a millimeter wave sensor, a radar laser sensor, an ultrasonic sensor,or others which may be fixedly mounted near a front grill or a frontbumper of the target vehicle. The distance measuring sensor 3 emits aradio wave, a light, or an ultrasonic wave toward the surroundings ofthe target vehicle, calculates the distance between an object around thetarget vehicle and the target vehicle, or the relative velocity of theobject around the target vehicle, based on the time to reception of itsreflected wave or the frequency difference of the reflected wave due toDoppler effect, and then outputs the calculated value to the controldevice 1.

The vehicle speed sensor 4 is a sensor for measuring a speed of a targetvehicle. For example, the vehicle speed sensor 4 may read out thevariation in a magnetic field generated by a magnet attached androtating with each wheel of the target vehicle as magnetic resistance bya MR (Magnetic Resistance) element. Then the vehicle speed sensor 4detects the magnetic resistance as a pulse signal proportional to therotating speed of each wheel so that it detects rotating speed of eachwheel and thus the speed of the target vehicle, and then outputs thedetected value to the control device 1.

The steering angle sensor 5 is a sensor for measuring a rotating angleof a steering shaft relating to a steering angle of a wheel of thetarget vehicle. For example, the steering angle sensor 5 may read out amagnetic resistance of a magnet embedded in the steering shaft by a MRelement, detect a rotating angle of the steering shaft, and then outputthe detected value to the control device 1.

The display device 6 is a device for displaying a variety ofinformation. For example, the display device 6 may be a liquid crystaldisplay which displays images taken by the imaging sensor 2 or a varietyof text messages.

The audio output device 7 is a device for audio outputting a variety ofinformation. For example, the audio output device 7 may be an in-carspeaker which audio outputs an audible alarm or an audio assist.

Next is an explanation of each part in the control device 1.

The feature point candidate extracting part 10 is a part for extractingfeature point candidates. In an example, the feature point candidateextracting part 10 may apply an image processing technique such as grayscale processing, digitizing processing, edge detecting processing, andso on to an image of a view around a target vehicle taken by the imagingsensor 2, and then extract feature point candidates.

In an example, the feature point candidate extracting part 10 mayextract for example a pixel whose luminance difference with respect toan adjacent pixel is greater than or equal to a predetermined value, ora center pixel of a pixel group whose luminance pattern matches apredetermined luminance pattern, and so on as a feature point candidate,and then store the position (coordinate value) of the feature pointcandidate (i.e. the extracted pixel) in the RAM.

The positional relation obtaining part 11 is a part for obtaining arelative positional relation between an actual position of a featurepoint candidate and an actual position of the target vehicle. In anexample, the positional relation obtaining part 11 obtains, based on anoutput of the distance measuring sensor 3, a distance between an actualposition of a feature point candidate extracted by the feature pointcandidate extracting part 10 and the distance measuring sensor 3 (i.e.the target vehicle) (hereinafter called “a space distance”), a directiondefined by an actual position of a feature point candidate relative tothe distance measuring sensor 3 (i.e. the target vehicle) as seen fromthe distance measuring sensor 3 (i.e. the target vehicle) (hereinaftercalled “an actual direction”), a velocity of an actual position of afeature point candidate relative to the distance measuring sensor 3(i.e. the target vehicle) (hereinafter called “a relative velocity”), orothers.

The positional relation obtaining part 11 stores the space distance, theactual direction and the relative velocity obtained for each featurepoint candidate in the RAM in relation to the coordinate value of eachfeature point candidate stored by the feature point candidate extractingpart 10.

The feature point selecting part 12 is a part for selecting a featurepoint from a feature point candidate(s). In an example, based on therelative velocity of the actual position of a feature point candidatewith respect to the distance measuring sensor 3 (i.e. the targetvehicle) obtained by the positional relation obtaining part 11, thefeature point selecting part 12 excludes a feature point candidate whoseactual position is on a moving body (for example, a pedestrian or amoving vehicle and so on) from a feature point candidate to be used in asubsequent process, and selects a feature point candidate whose actualposition is on a stationary body (for example, a road indication, atraffic sign, a guard rail, a traffic light, or a parked vehicle and soon) as a feature point candidate to be used in the subsequent process.

The reason for this is that selecting a feature point candidate whoseactual position is on a moving body as a feature point candidate to beused in a subsequent process makes it difficult to distinguish thedisplacement of the feature point from the displacement of the targetvehicle, which would reduce the accuracy of the estimation of the movingstate of the target vehicle.

Based on the space distance which is the distance between the actualposition of the feature point candidate and the distance measuringsensor 3 (i.e. the target vehicle) for example, the feature pointselecting part 12 selects as a first feature point among plural featurepoint candidates the feature point candidate whose space distance is thesmallest among the feature point candidates. Alternatively, the featurepoint selecting part 12 may select as the first feature point thefeature point candidate whose space distance is the largest among thefeature point candidates.

Alternatively, based on the actual direction which is the directiondefined by the actual position of the feature point candidate relativeto the distance measuring sensor 3 (i.e. the target vehicle) as seenfrom the distance measuring sensor 3 (i.e. the target vehicle), thefeature point selecting part 12 may select as the first feature pointthe feature point candidate whose actual direction relative to apredetermined direction is the closest to a predetermined angle and isgreater than or equal to the predetermined angle (for example, 0 degreesas in the case where the angle increases clockwise beginning at afrontward direction of the target vehicle as 0 degrees).

Based on the space distance of the first feature point and the spacedistances of other feature point candidates, the feature point selectingpart 12 selects as a second feature point among plural feature pointcandidates the feature point candidate whose difference between itsspace distance and the space distance of the first feature point is thesmallest among the feature point candidates and is greater than or equalto a predetermined value.

The feature point selecting part 12 continues the selection of featurepoints such as a third feature point, a fourth feature point, a fifthfeature point, and so on, until the feature point selecting part 12selects a necessary and sufficient number of feature points forgenerating homography.

Based on the actual direction of the first feature point and the actualdirection of other feature point candidates, the feature point selectingpart 12 may select as the second feature point among plural featurepoint candidates the feature point candidate whose angle between itsactual direction and the actual direction of the first feature point isthe smallest among the feature point candidates and is greater than orequal to a predetermined value.

Similarly, the feature point selecting part 12 selects the third featurepoint, the fourth feature point, the fifth feature point, and so on, inthe same way that the feature point selecting part 12 selects the secondfeature point, and continues the selection of feature points until thefeature point selecting part 12 selects a necessary and sufficientnumber of feature points for generating homography.

The feature point selecting part 12 may select as a feature point afeature point candidate for which both its space distance and its actualdirection meet the respective predetermined conditions. Alternatively,the feature point selecting part 12 may divide the image taken by theimaging sensor 2 into a plurality of small sections and select apredetermined number of feature points from feature point candidates ineach small section.

FIGS. 2A, 2B, and 2C are each an illustration of a method for selectinga feature point from a plurality of feature point candidates. FIG. 2Aillustrates that the moving state estimating device 100 extracts featurepoint candidates (“X” marks in FIG. 2A) on other vehicles V2 and V3,obtains the relative velocity of the actual position of each featurepoint candidate with respect to the target vehicle V1, and selects thefeature point candidate C2 of the parked vehicle V2 as the feature pointcandidate to be used in a subsequent process while excluding the featurepoint candidate C1 of the moving vehicle V3 as a feature point candidateto be used in the subsequent process.

FIG. 2B illustrates that the moving state estimating device 100 extractsfeature point candidates (“X” marks in FIG. 2B) on guard rails, obtainsthe space distances between the target vehicle V1 and each feature pointcandidate, selects as the first feature point P1 the feature pointcandidate whose space distance is the smallest among the feature pointcandidates, and then selects the second feature point P2 so that thedifference between the space distance D1 of the first feature point P1and the space distance D2 of the second feature point P2 is greater thanor equal to a predetermined value, and selects the third feature pointP3 so that the difference between the space distance D2 of the secondfeature point P2 and the space distance D3 of the third feature point P3is greater than or equal to the predetermined value. It will be assumedthat the fourth feature point (not shown in FIG. 2B) or later areselected similarly.

FIG. 2C illustrates that the moving state estimating device 100 extractsfeature point candidates (“X” marks in FIG. 2C) on road indications(traffic lane lines), obtains the actual direction of each feature pointcandidate (i.e. the direction defined by an actual position of thefeature point candidate relative to the target vehicle V1 as seen fromthe target vehicle V1), selects as the first feature point P1 thefeature point candidate whose actual direction relative to a frontwarddirection of the target vehicle is the closest to 0 degrees among thefeature point candidates and is greater than or equal to 0 degrees, andthen selects the second feature point P2 so that the angle α between theactual direction of the first feature point P1 and the actual directionof the second feature point P2 is greater than or equal to apredetermined value, and selects the third feature point P3 so that theangle β between the actual direction of the second feature point P2 andthe actual direction of the third feature point P3 is greater than orequal to a predetermined value. It will be assumed that the fourthfeature point (not shown in FIG. 2C) or later are selected similarly.

The feature point selecting part 12 selects a necessary and sufficientnumber of feature points for calculating homography. The feature pointselecting part 12 stores for each feature point the luminance pattern ofthe pixel group (5×5 pixels for example) in which the feature pointexists, in the RAM so that, as will be described hereinafter, thecorresponding point extracting part 13 can extract a point correspondingto the feature point from a later image by use of a pattern matchingprocess.

The corresponding point extracting part 13 is a part for extracting acorresponding point. In an example, the corresponding point extractingpart 13, by using a pattern matching process, extracts a correspondingpoint for each feature point selected by the feature point selectingpart 12 by means of matching a luminance pattern in the later image(hereinafter called “the second image”, the second image is an imagetaken by the imaging sensor 2 after a predetermined time interval haspassed since the first image had been taken) which is different from theimage that had been taken by the imaging sensor 2 and used by thefeature point candidate extracting part 10 for extracting the featurepoint candidates (hereinafter called “the first image”), with theluminance pattern relating to the feature point stored in the RAM by thefeature point selecting part 12.

In an example, the corresponding point extracting part 13 may restrictthe range in the image which is to be subjected to the pattern matchingprocess, for example, by roughly estimating a moving state of thevehicle based on the outputs of the vehicle speed sensor 4 and thesteering angle sensor 5. This is to improve the efficiency of thepattern matching process.

The corresponding point extracting part 13 extracts a correspondingpoint (a corresponding point corresponds to a feature point and iscalled “a primary corresponding point” herein in order to differentiateit from a later-mentioned secondary corresponding point), and thecorresponding point extracting part 13 stores the luminance pattern ofthe pixel group (5×5 pixels for example) in which the primarycorresponding point exists, in the RAM so that the corresponding pointextracting part 13, by using a pattern matching process, can extract asubsequent corresponding point (the secondary corresponding point) whichcorresponds to the primary corresponding point from still a later image.

By tracking a feature point in sequence according to an order of, forexample, a feature point, a primary corresponding point, a secondarycorresponding point, a tertiary corresponding point, etc., the movingstate estimating device 100 can continuously estimate the moving stateof the target vehicle beginning at the time when the moving stateestimating device 100 obtains the image used for the selection of thefeature point.

The moving state estimating part 14 is a part for estimating a movingstate of a target vehicle. In an example, the moving state estimatingpart 14 generates homography by using the two-dimensional coordinate ofthe feature point selected by the feature point selecting part 12 andthe two-dimensional coordinate of the corresponding point extracted bythe corresponding point extracting part 13. The moving state estimatingpart 14 estimates translation or rotation of the target vehicle madeduring a time period from the time point when the imaging sensor 2 hadtaken the first image used for the extraction of the feature pointcandidate by the feature point candidate extracting part 10 until thetime point when the imaging sensor 2 has taken the second image used forthe extraction of the corresponding point by the corresponding pointextracting part 13.

By initiating estimation of a moving state of a target vehicle when thecontrol device 1 detects an obstacle (a curbstone, a concrete block wallor others for example) within a predetermined range from the targetvehicle based on an output of the distance measuring sensor 3, thecontrol device 1 can determine, for example, whether there is a chancethat the target vehicle may come into contact with the obstacle based onthe estimated moving state of the target vehicle even if the obstacledeviates from the imaging area of the imaging sensor 2 when the targetvehicle turns left or right.

If the control device 1 determines that there is a high probability thatthe target vehicle will come into contact with an obstacle, the controldevice 1 invites a driver's attention by making the display device 6display a warning message or by making the audio output device 7 outputa warning sound.

Based on the estimated moving state of the target vehicle, the controldevice 1 may make the display device 6 display a quasi-image whichrepresents the positional relation between the target vehicle and theobstacle, and may assist the driving of the target vehicle in order toavoid contact with the obstacle.

With reference to FIG. 3, the process by which the moving stateestimating device 100 estimates the moving state of the target vehicle(hereinafter called “the moving state estimating process”) is explainedin detail. FIG. 3 is a flowchart illustrating the flow of the movingstate estimating process. The control device 1 of the moving stateestimating device 100 executes the moving state estimating processrepeatedly, for example, after the blinker has been manipulated, untilthe vehicle completes the left/right turn.

Alternatively, the moving state estimating device 100 can execute themoving state estimating process repeatedly in case that the moving stateestimating device 100 detects the situation that there is a highprobability that the target vehicle may come into contact with anobstacle such as when the speed of the target vehicle is less than apredetermined value, or when the shift position is set to “R (Reverse)”position, or others.

First, the control device 1 obtains the image data taken by the imagingsensor 2 at a time point T1 (Step S1), and then by use of the featurepoint candidate extracting part 10, applies image processing to theimage data to extract feature point candidates, a pixel extracted beingone whose luminance difference with respect to an adjacent pixel isgreater than or equal to a predetermined value (Step S2).

Next, by use of the positional relation obtaining part 11, the controldevice 1 obtains the positional relation between the actual position ofeach feature point candidate and the position of the distance measuringsensor 3 based on the output value from the distance measuring sensor 3(Step S3). The control device 1 stores the positional relation (i.e. therelative velocity, the space distance and the actual direction) of eachfeature point candidate in relation to the coordinate value of thefeature point candidate in the RAM.

Next, by use of the feature point selecting part 12, the control device1 determines for each feature point candidate whether the actualposition of the feature point candidate remains stationary or is inmotion based on the relative velocity of the actual position of thefeature point candidate obtained by the positional relation obtainingpart 11, the speed of the target vehicle outputted by the vehicle speedsensor 4, and the steering angle (traveling direction) of the targetvehicle outputted by the steering angle sensor 5. By use of the featurepoint selecting part 12, the control device 1 excludes a feature pointcandidate whose actual position is on a moving body from a feature pointcandidate to be used in a subsequent process, while keeping a featurepoint candidate whose actual position is on a stationary body as afeature point candidate to be used in the subsequent process (Step S4).

Next, by use of the feature point selecting part 12, the control device1 selects as a first feature point the feature point candidate whosespace distance is the smallest among the space distances of the featurepoint candidates obtained by the positional relation obtaining part 11(Step S5).

Next, after setting the space distance of the actual position of thefirst feature point as the basis, by use of the feature point selectingpart 12, the control device 1 selects other feature points in sequenceso that the difference between each space distance is greater than orequal to a predetermined value (Step S6).

The control device 1 selects a necessary and sufficient number offeature points for calculating homography, by use of the feature pointselecting part 12, and stores for each selected feature point theluminance pattern of the pixel group (5×5 pixels for example) in whichthe feature point exists, in the RAM so that a corresponding point whichcorresponds to the feature point may be extracted from an image takensubsequently (Step S7).

Next, the control device 1 obtains image data taken by the imagingsensor 2 at a time point T2 (the time point T2 is later than the timepoint T1) (Step S8). Then, by use of the corresponding point extractingpart 13, the control device 1 applies the same image processing as inStep S1 to the image data taken at the time point T2 and, with respectto the luminance pattern stored for each feature point, restricts therange in the image, which is to be subjected to a pattern matchingprocess, based on the outputs of the vehicle speed sensor 4 and thesteering angle sensor 5 (Step S9). This is to improve the efficiency ofthe pattern matching process. In addition, the corresponding pointextracting part 13 may restrict the range in the image which is to besubjected to the pattern matching process based on the coordinate ofeach feature point, or may perform the pattern matching process withoutrestricting the range in the image.

Next, by use of the corresponding point extracting part 13, the controldevice 1 performs the pattern matching process for each restricted rangeof the image by matching the luminance pattern in the restricted rangewith the respective luminance pattern of the feature point, and extractsthe center pixel of the matched luminance pattern as the point whichcorresponds (i.e. the “corresponding point”) to the feature point (StepS10).

Lastly, by use of the moving state estimating part 14, the controldevice 1 calculates homography based on the two-dimensional coordinateof the respective feature point selected by the feature point selectingpart 12 and the two-dimensional coordinate of the respectivecorresponding point extracted by the corresponding point extracting part13. Then the control device 1 estimates the state of translation androtation of the target vehicle made between the time point T1 and thetime point T2 (Step S11). Then the control device 1 terminates thismoving state estimating process.

In this way, the moving state estimating device 100 selects featurepoints whose actual positions are not biased in terms of distance anddirection while combining the output of the imaging sensor 2 and theoutput of the distance measuring sensor 3, and calculates homographybased on the coordinates of those feature points and the coordinates ofthe corresponding points which correspond to those feature points. Thus,the moving state estimating device 100 can estimate the moving state ofthe imaging sensor 2 and therefore the moving state of the targetvehicle with a high degree of accuracy.

This is because, when there is a bias in the actual positions of thefeature points, the moving state estimating device 100 becomes easilyaffected by noises which come to be mixed in while selecting thosefeature points and which arise from the vibration of the target vehicleor the photographic subject, the error in the image processing orothers, and because the moving state estimating device 100 can alleviatethe effect of those noises by distributing the actual positions of thefeature points in terms of the distance and the direction.

Additional Statement

Although the present invention has been described above with respect topreferable embodiments, the present invention is not to be thus limited,and the above-described embodiments are to receive various modificationsand substitutions without departing from the scope of the presentinvention.

For example, in the embodiments above, although the moving stateestimating device 100 obtains the space distance, the actual direction,and the relative velocity of the actual position of the feature pointcandidate based on the output of the distance measuring sensor 3, themoving state estimating device 100 may obtain the space distance, theactual direction, and the relative velocity of the actual position ofthe feature point candidate based on the output of a stereo cameracapable of measuring a distance to a photographic subject. In such case,the moving state estimating device 100 can omit the distance measuringsensor 3.

In addition, in the embodiments above, although the moving stateestimating device 100 estimates the moving state of the vehicle, it mayestimate the moving state of other moving bodies such as an industrialrobot, a vessel, a motorcycle, an aircraft, and so on. In such case, themoving state estimating device 100 may estimate the moving state in thevertical direction in addition to the moving state in the plane surfacefrom front to back and from side to side.

Moreover, in the embodiments above, although the corresponding pointextracting part 13 employs the pattern matching process to extract thepoint which corresponds (i.e. the “corresponding point”) to the featurepoint, the corresponding point extracting part 13 may employ any othertechnique as far as it can extract the corresponding point. For example,the corresponding point extracting part 13 may extract the correspondingpoint with an optical flow technique.

The present application claims priority from Japanese Patent ApplicationNo.2007-332275 filed on Dec. 25, 2007, which is incorporated herein byreference.

1. A moving state estimating device comprising: a feature pointcandidate extracting part configured to extract feature point candidatesin an image taken at a first time point by an imaging sensor mounted ona moving body; a positional relation obtaining part configured to obtaina positional relation of an actual position of each feature pointcandidate relative to the moving body; a feature point selecting partconfigured to select a feature point candidate as a feature point basedon the positional relation; a corresponding point extracting partconfigured to extract a corresponding point in an image taken by theimaging sensor at a second time point corresponding to the selectedfeature point; and a moving state estimating part configured to estimatea moving state of the moving body from the first time point to thesecond time point based on a coordinate of the selected feature pointand a coordinate of the corresponding point.
 2. A moving stateestimating device according to claim 1, wherein the positional relationincludes a distance between the actual position of the feature pointcandidate and the moving body, a direction of the actual position of thefeature point candidate as seen from the moving body, or a relativevelocity of the actual position of the feature point candidate withrespect to the moving body.
 3. A moving state estimating deviceaccording to claim 1, wherein the feature point selecting part isconfigured to determine whether the feature point candidate isstationary or not, and to select the feature point from stationaryfeature point candidates.
 4. A moving state estimating device accordingto claim 1, wherein the feature point selecting part is configured toselect feature points so that a distance difference between an actualposition of each feature point and the moving body is greater than orequal to a predetermined value.
 5. A moving state estimating deviceaccording to claim 1, wherein the feature point selecting part isconfigured to select feature points so that angles between directions ofactual positions of each feature point as seen from the moving body aregreater than or equal to a predetermined value.